{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.检验指标确定（10分）\n",
    "\n",
    "banner的目的是让用户点击banner进入双十二主会场,所以二类指标是**点击率**.\n",
    "\n",
    "做广告不能显著降低用户体验,所以应将用户粘性指标(例如**人均活跃时长**)作为一类指标.\n",
    "\n",
    "## 2.确定检验统计量 （5分）\n",
    "\n",
    "定义统计口径,1名用户点击banner记为1,未点击记为0,则检验统计量可以取**样本平均数**(这样得到的样本平均数就是点击率).\n",
    "\n",
    "## 3.埋点收集数据 （10分）\n",
    "\n",
    "| 属性名       | 属性说明                 |\n",
    "| ------------ | ------------------------ |\n",
    "| banner_color | 返回当前banner的颜色     |\n",
    "| user_id      | 返回用户id               |\n",
    "| banner曝光   | 返回banner是否向用户曝光 |\n",
    "| 点击banner   | 用户是否点击banner       |\n",
    "| 进入时间     | banner向用户曝光的时间戳 |\n",
    "\n",
    "## 4.确定H0,H1 （10分）\n",
    "\n",
    "一类指标： \n",
    "\n",
    "H0: 实验组人均活跃时长 - 对照组人均活跃时长 = 0\n",
    "\n",
    "H1: 实验组人均活跃时长 - 对照组人均活跃时长不等于0\n",
    "\n",
    "二类指标 : \n",
    "\n",
    "H0: 实验组点击率 - 对照组点击率 = 0\n",
    "\n",
    "H1: 实验组点击率 - 对照组点击率 不等于0\n",
    "\n",
    "(思路是先看有没有显著性差异,再去比较是变好还是变差)\n",
    "\n",
    "## 5.确定显著水平α （5分)\n",
    "\n",
    "在借助软件做具体计算的前提下,我更赞成直接计算p值,而不用事先确定一个α.\n",
    "\n",
    "![image-20210104000943387](https://cdn.jsdelivr.net/gh/Wi2077/image/image-20210104000943387.png)\n",
    "\n",
    "## 6.计算样本量 （15分）\n",
    "\n",
    "根据前面建立的假设,应选择均值之差双侧检验的样本量计算公式:\n",
    "\n",
    "![image-20210104015621860](https://cdn.jsdelivr.net/gh/Wi2077/image/image-20210104015621860.png)\n",
    "\n",
    "上式中的σ暂时无法得出.故使用网上的AB test计算器来估算样本量,需提供以下4个参数:\n",
    "\n",
    "Statistical power（1-**β**）: 取1-20%=80%\n",
    "\n",
    "Statistical Significance（1-**α**）: 取 95%\n",
    "\n",
    "Baseline rate（conversation rate）: 在我们的实验里，baseline就是旧方案的点击率, 取10%\n",
    "\n",
    "Minimum Detectable Effect: 这个参数衡量了我们对实验的判断精确度的最低要求. 取5%\n",
    "\n",
    "计算结果如下:\n",
    "\n",
    "![image-20210104024738161](https://cdn.jsdelivr.net/gh/Wi2077/image/image-20210104024738161.png)\n",
    "\n",
    "## 7 利用统计工具实现检验(45分)\n",
    "\n",
    "简单处理数据发现,group和landing page存在交叉情况:\n",
    "\n",
    "![image-20210104004131280](https://cdn.jsdelivr.net/gh/Wi2077/image/image-20210104004131280.png)\n",
    "\n",
    "考虑业务逻辑,后续计算只对比control+old_page和treatment+new_page两组数据.\n",
    "\n",
    "数据user_id有重复记录,只保留同一用户的最后一次记录."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp      group landing_page  converted\n",
       "0   851104  2017-01-21 22:11:48.556739    control     old_page          0\n",
       "1   804228  2017-01-12 08:01:45.159739    control     old_page          0\n",
       "2   661590  2017-01-11 16:55:06.154213  treatment     new_page          0\n",
       "3   853541  2017-01-08 18:28:03.143765  treatment     new_page          0\n",
       "4   864975  2017-01-21 01:52:26.210827    control     old_page          1"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# 读取数据，查看前5行\n",
    "df = pd.read_csv(r'C:\\Users\\weiha\\Downloads\\ab_data.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 移除错误数据(control+new page和treatment+old page)\n",
    "df2 = df[~((df.landing_page == \"new_page\")&(df.group == \"control\"))]\n",
    "df3 = df2[~((df2.landing_page == \"old_page\")&(df2.group == \"treatment\"))]\n",
    "# 去除user id重复的行\n",
    "df4 = df3.drop_duplicates(subset=[\"user_id\"],keep=\"last\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.1203863045004612"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# control组的点击率\n",
    "control_converted = df4.query('group==\"control\"').converted.mean()\n",
    "control_converted"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11880806551510564"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# treatment组的点击率\n",
    "treatment_converted = df4.query('group==\"treatment\"').converted.mean()\n",
    "treatment_converted"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "AB Test封装函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "import statsmodels.stats.proportion as ssp\n",
    "\n",
    "def abtest(df: pd.DataFrame, alpha=0.05, group_col: str = None, value_col: str = None):\n",
    "    '''\n",
    "    :param df: 被分析DateFrame对象\n",
    "    :param alpha: 临界值\n",
    "    :param group_col: 组列的名字，默认为df的第一列\n",
    "    :param value_col: 值列的名字,默认为df的第2列\n",
    "    :return:best_group_name,pdf\n",
    "        best_group_name:最优组\n",
    "        pdf:最优组与其他组的差异性\n",
    "    '''\n",
    "    # 列名\n",
    "    if not group_col:\n",
    "        group_col = df.columns[0]\n",
    "    if not value_col:\n",
    "        value_col = df.columns[1]\n",
    "\n",
    "    # 寻找最优组与最优质值\n",
    "    best_group_name = df.groupby(group_col)[value_col].mean().sort_values(ascending=False).index.tolist()[0]\n",
    "    best_group_values = df[df[group_col] == best_group_name][value_col]  # 最优组的values\n",
    "    # 去除最优组的组名\n",
    "    group_names = df[group_col].unique().tolist()\n",
    "    group_names.remove(best_group_name)\n",
    "    # 初始化返回数据\n",
    "    pdf = pd.DataFrame(columns=[group_col,'mean', 'pvalue', 'ptype'])\n",
    "    \n",
    "    converted_baseline = best_group_values.sum()\n",
    "    n_baseline = len(best_group_values)\n",
    "    \n",
    "    for group_name in group_names:\n",
    "        group_values = df[df[group_col] == group_name][value_col]\n",
    "        \n",
    "        # 独立双样本，样本大小n＞30，总体均值和标准差未知，采用Z检验\n",
    "        converted_group = group_values.sum()\n",
    "        n_group = len(group_values)\n",
    "        \n",
    "        z_score, pvalue = ssp.proportions_ztest([converted_baseline,converted_group],[n_baseline,n_group], alternative = \"two-sided\")\n",
    "        \n",
    "        if pvalue >= alpha:\n",
    "            ptype = \"无显著差异\"\n",
    "        else:\n",
    "            ptype = \"有显著差异\"\n",
    "        # 添加数据\n",
    "        pdf.loc[pdf.shape[0]] = {group_col: group_name,'mean':group_values.mean(),  'pvalue': pvalue, 'ptype': ptype}\n",
    "\n",
    "    return best_group_name,best_group_values.mean(), pdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('old_page',\n",
       " 0.1203863045004612,\n",
       "   landing_page      mean    pvalue  ptype\n",
       " 0     new_page  0.118808  0.189883  无显著差异)"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    " abtest(df4, 0.05, 'landing_page', 'converted')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "两个banner的点击转化率无显著性差异,需要重新调整策略."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
